Focal plane coding for digital imaging

ABSTRACT

An imaging system includes an array of lenses, a plurality of sensor pixels for each lens, the sensor pixels being on an image plane of the imaging system, and a corresponding plurality of focal plane coding elements. A focal plane coding element for each sensor pixel has multiple sub-pixel resolution elements. The focal plane coding element being between the lens and each sensor pixel, wherein sub-pixel resolution elements over the plurality of focal plane coding elements represent a selected transform matrix having a non-zero determinant. The output of the plurality of sensor pixels being an image multiplied by this matrix

BACKGROUND

Imaging system design begins with the focal plane. Assume the focalplane has pixels of a size p, with a size of the image on the focalplane being d. The number n of pixels is n=d/p. The N.A. or numericalaperture roughly determines the field of view of the system, and isdefined as N.A.=n₀ sin θ, where n₀ is the refractive index of the mediumthrough which the light has traveled. Assuming the medium is air and thesmall angle approximation is valid, n₀≈1, so N.A.=sin θ. The angularresolution is Δθ_(p)=p/f due to the focal plane and Δθ_(λ)=λ/d due tothe diffraction limit. Since f and d are related by the N.A.,Δθ_(p)/Δθ_(λ)=N.A.p/λ. Thus, for a given focal plane, the angularresolution is inversely proportional to f, meaning that the thicker thesystem, the better the angular resolution. Additionally, the size andnumber of pixels determine the spatial resolution.

If p could be reduced by a desired scaling factor, then f could bereduced by an order of magnitude, while maintaining the resolution ofthe system. However, there are limits on the ability to enhance imagingby simply increasing the density of pixels. Further, post-processorscannot process information that has not been captured by the imagingsystem. Additionally, increased optical performance often meansincreased complexity.

Thus, techniques other than simply increasing pixel density and relianceon post processing are needed to advance imaging systems. Desiredadvances include reducing camera thickness, improving resolution andimproving data efficiency.

Current attempts to achieve these advances include integratedcomputational imaging systems (ICIS). The design of ICIS simultaneouslyconsiders optics, optoelectronics and signal processing, rather thanindependently designing the optics. System performance for the ICIS isrealized through joint optimization of optics, focal planeoptoelectronics and post-detection algorithms. The computational imagingused to balance processing between optics and electronics are typicallyclassified into three categories: wavefront encoding, multiplex imagingand feature extraction. Wavefront encoding involves modifying thewavefront phase at or near the pupil plane of an imaging system. Inmultiplex imaging, typically the optics introduce redundant informationused in post-processing detection. In direct feature extraction, featureextraction estimates are made of transform coefficients that are thenused to make a decision. Often, all three categories are employed.

Typically, ICIS use non-focal sensors, e.g., interferometric systems,wavefront coded systems. The purposeful blurring attendant with suchnon-focal sensors, which is then removed in post-processing, providesmultiplexing in the optical field. However, this blurring does notexploit the point of high field entropy of the system. For conventionalimaging of remote objects, highest entropy is at the focal plane. Thus,rather than using the information inherent in one-to-one mapping, i.e.,that there is a relationship between spatially separated regions, thedetectors of these systems are acting as pixel sensors rather than imagesensors.

SUMMARY OF THE INVENTION

It is a feature of an embodiment of the present invention to provide animaging system having sub-pixel resolution. It is another feature of anembodiment of the present invention to provide multiplexing while stillanalyzing a true image. It is another feature of an embodiment of thepresent invention to provide imaging systems of reduced thickness whilemaintaining resolution and/or to improve resolution and data efficiency.

At least one of the above and other features may be realized by applyingICIS at the focal plane of the imaging system.

At least one of the above and other features may be realized bymultiplexing in the electrical plane.

At least one of the above and other features may be realized byproviding an imaging system including an array of lenses, a plurality ofsensor pixels for each lens, the sensor pixels being on an image planeof the imaging system, and a corresponding plurality of focal planecoding elements. A focal plane coding element for each sensor pixel hasmultiple sub-pixel resolution elements. The focal plane coding elementbeing between the lens and each sensor pixel, wherein sub-pixelresolution elements over the plurality of focal plane coding elementsrepresent a selected transform matrix having a non- zero determinant.The output of the plurality of sensor pixels being an image multipliedby this matrix.

The filter may provide sub-pixel shifted multiple images on each sensorpixel. The focal plane coding element may be an apertured mask. Theimaging system may include color filters, which may be integral with thefocal plane coding element. A birefringent structure may be adjacent thefocal plane coding element. A corresponding plurality of focusing lensesmay be included between the focal plane encoding element and acorresponding sensor pixel. The selected transform matrix has fewer rowsthan columns. At least one sensor pixel receives light from more thanone lens of the array of lenses.

A processor receiving the outputs of the sensor pixels and multiplyingthe outputs by an inverse matrix may be included. The processor mayreconstruct an image from the outputs, a number of image pixels in theimage being greater than the plurality of sensor pixels.

At least one of the above and other features may be realized byproviding an imaging system including an array of lenses, a plurality ofsensor pixels for each lens, a corresponding plurality of filters, and aprocessor. A filter for each sensor pixel has multiple sub-pixelresolution elements and provides a sub-pixel shifted multiple image oneach sensor pixel. The processor receives outputs from each sensor pixeland reconstructs an image, a number of image pixels in the image beinggreater than the plurality of sensor pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become readily apparent to those of skill in the art by describingin detail embodiments thereof with reference to the attached drawings,in which:

FIG. 1 is a schematic side view of an array of micro-cameras accordingto an embodiment of the present invention;

FIG. 2 is an example of a mask to be used as a focal plane codingelement in accordance with an embodiment of the present invention; and

FIG. 3 is a schematic side view of an array of micro-cameras accordingto another embodiment of the present invention.

DETAILED DESCRIPTION

U.S. Provisional Application Ser. No. 60/538,506 filed Jan. 26, 2004 andentitled “Focal Plane Coding for Digital Imaging” is herein incorporatedby reference in its entirety for all purposes.

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, in which preferred embodimentsof the invention are shown. The invention may, however, be embodied indifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the concept of the invention to those skilled in the art. In thedrawings, the thickness of layers and regions are exaggerated forclarity. It will also be understood that when a layer is referred to asbeing “on” another layer or substrate, it may be directly on the otherlayer or substrate, or intervening layers may also be present. Further,it will be understood that when a layer is referred to as being “under”another layer, it may be directly under, or one or more interveninglayers may also be present. In addition, it will also be understood thatwhen a layer is referred to as being “between” two layers, it may be theonly layer between the two layers, or one or more intervening layers mayalso be present. Like numbers refer to like elements throughout.

FIG. 1 illustrates a multiple aperture imaging system 10. Each apertureincludes an imaging lens 12, a focal plane coding element 14 and adetector plane 16. The detector plane 16 is located at or near the imageplane of the system 10. The detector plane 16 for each aperture includesa plurality of pixels in accordance with a size of an image at the imageplane and the selected detector size.

In addition to the imaging lens 12, each aperture may include aplurality of substrates 22, 24, which may be separated by a spacer 26.The spacer 26 may be integral with one of the substrates 22, 24 or maybe formed on a separate spacer substrate. In the particular embodimentshown, a first refractive lens 18 is on a top surface of the substrate22, a diffractive lens is on a bottom surface of the substrate 22 andthe imaging lens 12 is on a top surface of the substrate 24. Thediffractive lens 20 may correct for aberrations.

Here, the focal plane coding element 14 is provided on or adjacent to abottom surface of the substrate 24 and the detector plane 16 is providedon or adjacent to the focal plane coding element 52. Additionalsubstrates providing additional surfaces with optical power therein maybe provided in a similar fashion as dictated by the requirements of thesystem 10. The substrates 22 and 24 may have surfaces parallel with oneanother and may be bonded together across the system 10 as shown in FIG.1.

The focal plane coding element 14 samples and/or remaps the focusedimage for coded multiplexing at the optical-electronic interface. Theoutputs of the detector plane are provided to a processor 30, whichelectrically process the information as discussed below. Here, there isno multiplexing of the optical field, since the image is isomorphic andwell focused at the focal plane. Rather, multiplexing occurs from thepixels sampling multiple optical resolution elements. The focal planecoding is to maintain the focal distribution while remapping samplingacross the image plane to enable digital reconstruction of the image.

In a conventional imaging system, the focal plane averages wavelengthscale features within each pixel. A pixel measurement m may be modeledas in Equation (1): $\begin{matrix}{m = {\int_{A}{{I(r)}\quad{\mathbb{d}r}}}} & (1)\end{matrix}$where A is the area of the pixel and I is the intensity.

With compressive coding of the present invention, the pixel measurementis given by Equation (2):m _(ij) ∫p _(ij)(r)I(r)dr  (2)where p_(ij)(r) is is the focal plane code in the focal plane codingelement 14 for the i^(th) pixel in the j^(th) aperture of themulti-aperture system 10.

Compressive imaging uses focal plane coding to allow reconstruction ofimages having a higher spatial resolution than the number of pixels,i.e., detector elements, on the focal plane. In other words, if thenumber of image pixels is greater than the number of physical pixels. Ifthe image is broken into sub-blocks, a linear transformation as shown inEquation (3) may be implemented on each sub-block:m=Hs  (3)where s is the source or actual image and H is a rectangular transform.The rectangular transform may be non-local and non-convex. Differentknown rectangular transforms may be used, such as Hadamard, discretecosine and quantized cosine, in which the discrete consine transform isrounded to correspond to a value in the set of the quantized cosinetransform.

These transforms provide multiple lower resolution, i.e., pixel level,shifted with sub-pixel accuracy which are detected to recover a singlehigher resolution, i.e., sub-pixel level, image. Thus, the multiplexingis done at the electrical plane, since each detector samples multipleresolution elements.

In accordance with the present invention, the outputs of the detectorplane 16 to a processor or electrical plane 30 are not the image itself,but a matrix multiple of the image. Any appropriate matrix may be used,as long as the determinant of the matrix is not zero. Preferably, thematrix will allow as much power as possible on each pixel through to thedetector plane 16. The processor may then take the inverse matrixtransform to retrieve the image.

In the system 10 as shown in FIG. 1, the focal plane coding element maybe created having the focal plane code p_(ij)(r) in accordance with H.An inverse transform H⁻¹ is applied by the processor 30 to themeasurement m detected at the detecting plane 16 to get the estimatedimage. The difference between the estimated image and the real image isthe noise of the system 10.

Different mechanisms may be used to realize physical implementation ofthe focal plane coding element 14 to provide multiple optical resolutionelements for each detector. One such mechanism is coded masks created inaccordance with the transform matrix selected. Such masks may includeopaque amplitude patterns on optical transparent materials. Thesepatterns may also be combined with surface relief structures to provideboth phase and amplitude mapping. Breaking the image into sub-blocks,the transformation may be implemented on each sub-block.

An example of a mask pattern for a 4×4 sub-block is shown in FIG. 2. Theparticular transform H in this example is a Hadamard-S matrix, which isa [0,1] matrix obtained by shifting the Hadamard matrix elementwise upby one and scaling it by half. As shown in FIG. 2, a pixel filter 40 ofthe focal plane coding element 14, here a mask pattern, corresponding toeach pixel is segmented into a 4×4 grid of optical resolution elementsor apertures 42, where a value of one is represented as white and avalue of zero is represented as black.

Actually, FIG. 2 shows the mask pattern for a particular matrix forEquation (3). Each 4×4 sub-pattern is placed over each pixel. Withineach micro-camera, each pixel receives a different portion of the image.Between micro-cameras, each corresponding receives identical or nearlyidentical images. FIG. 2. shows the mask patterns on each of thesecorresponding pixels over the camera array. That is, each of thesecorresponding pixels in different micro-cameras, that receive nearidentical images, will be multiplied a different 4×4 sub-pattern. Thismay be most readily achieved by providing a same sub-pattern for eachpixel in a given micro-camera. Over the array of micro-camera, eachpixel may be multiplied by each 4×4 sub-pattern.

As can be seen in FIG. 2, all incident optical power is transmitted ontoone of the pixels. Most of the pixels have half or more of the powertransmitted thereon. Each optical resolution element 42 will multiplythe image with the matrix component for that optical resolution element42.

Ideally, the amount of power incident on each pixel should be maximized.This may be realized by examining the pixel with least amount of powerthereon and altering the matrix to optimize this pixel while stillkeeping the determinant non-zero. Such optimization may be iterative.

In the example shown in FIG. 2, each pixel receives light from an area40 including a plurality, here 4×4, optical resolution elements 42. Moregenerally, each pixel has a size p_(x)p_(y), and each pixel filter 40has a size d_(x)d_(y) with a plurality of m_(x)m_(y) optical resolutionelements 42 of size q_(x)q_(y). Normally, such a camera would have aresolution of 1/p_(x)×1/p_(y). The size of the pixel region 40d_(x)d_(y) may be the same as or greater than the pixel size P_(x)P_(y).Thus, the power incident on each pixel is equal to a summation, over alloptical resolution elements 42 on the corresponding pixel filter 40, ofthe power on each optical resolution elements 42 multiplied by thetransmittance of the optical resolution elements 42. In this case, theresolution of the camera may now be as low as 1/q_(x)×1/q_(y)

Additional distribution of the image onto the pixels may be combinedwith these masks to allow remapping of the image. For example,separating of the image into vertical and horizontal polarization modescan be used to create coding elements that displace focal images acrossthe image plane without diffraction. If a suitable birefringent materialis available in the wavelength region of interest, this material may beused for the mask. If this is not practical, subwavelength structurespresenting different effective indices of refraction for the verticaland horizontal polarization states may be incorporated.

Color images may be realized with the system 10 by placing differentcolor filters in the path of different elements in the detection plane16. Typically, three color filters, e.g. red, green and blue, arerequired, so each color filter may be in a path of a third of themacro-pixels. Since the eye is more sensitive to green, in many casesmore green filters are used than red and blue, e.g. 25% of the pixelshave blue and red filters and 75% have green filters Alternatively,color sensitive rather than binary absorption masks may be used as thecoding elements in the focal plane coding element 16. Since a colorfilter for a given wavelength will absorb most of the other wavelengths,this color filter can serve as providing a value of one (or hightransmittance) for the design wavelength and a value of zero (or lowtransmittance) for other wavelengths.

Another imaging system 50 is shown in FIG. 3. Here, sensors in adetector plane 56 are too far from the focal plane coding element 14 toreceive all of the light transmitted therefrom. In order to addressthis, another array of lenses 52 is provided for imaging the output ofthe focal plane coding element 14 onto the sensors in the detector plane56.

While the above coding has assumed that all measurements are used, i.e.,a non-compressive design, compressive coding may also be used. Incompressive system design, unimportant coefficients of the coefficientvector m are discarded, thus turning the corresponding row in thetransform matrix H to 0. Therefore, fewer pixels may need to beemployed, i.e., no physical implementation of these pixels for thediscarded terms is required. Thus, the number of electronicallygenerated image pixels is greater than the number of physical pixels.The compression ratio is defined by the number of rows to the number ofcolumns in the transform matrix

Further, the above coding has assumed a transform matrix having elementsfrom the set (1, 0). Transform matrices having elements from other sets,e.g., (1, −1, 0) may be used. To approximate a (1, −1, 0) matrix, athreshold t is established. Any element having a value greater than t isset to 1, any element having a value less than −t is set to −1 and allothers are set to 0. Any threshold with an acceptable condition numberfor the matrix transform may be used. Such matrices may be implementedusing photonic crystals or a combination of amplitude and phase masks.

Thus, focal plane coding uses optical elements to encode digital imagingsystems such that the spatial resolution of the reconstructed imageexceeds the nominal spatial resolution of the electronic focal plane.Focal plane coding may be used to make cameras thinner than existingcameras while maintaining resolution and/or to improve resolution anddata efficiency of digital imaging systems.

Embodiments of the present invention have been disclosed herein and,although specific terms are employed, they are used and are to beinterpreted in a generic and descriptive sense only and not for purposeof limitation. Accordingly, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madewithout departing from the spirit and scope of the present invention asset forth in the following claims.

1. An imaging system, comprising: an array of lenses; a plurality ofsensor pixels for each lens, the sensor pixels being on an image planeof the imaging system; and a corresponding plurality of focal planecoding elements, a focal plane coding element provided for each sensorpixel having multiple sub-pixel resolution elements, the focal planecoding element being between the lens and sensor pixel, whereinsub-pixel resolution elements over the plurality of focal plane codingelements represent a selected transform matrix, the output of theplurality of sensor pixels being an image multiplied by the selectedtransform matrix, the selected transform matrix having a non-zerodeterminant.
 2. The imaging system as recited in claim 1, wherein thefocal plane coding element provides sub-pixel shifted multiple images oneach sensor pixel.
 3. The imaging system as recited in claim 1, whereinthe focal plane coding element is an apertured mask.
 4. The imagingsystem as recited in claim 1, further comprising color filters.
 5. Theimaging system as recited in claim 1, wherein the color filters areintegral with the focal plane coding element.
 6. The imaging system asrecited in claim 1, further comprising a birefringent structure adjacentthe focal plane coding element.
 7. The imaging system as recited inclaim 1, further comprising a corresponding plurality of focusinglenses, a focusing lens between the focal plane encoding element and acorresponding sensor pixel.
 8. The imaging system as recited in claim 1,wherein the selected transform matrix has fewer rows than columns. 9.The imaging system as recited in claim 1, wherein at least one sensorpixel receives light from more than one lens of the array of lenses. 10.The imaging system as recited in claim 1, further comprising a processorreceiving the outputs of the sensor pixels and multiplying the outputsby an inverse of the selected transform matrix.
 11. The imaging systemas recited in claim 10, wherein the processor reconstructs an image fromthe outputs, a number of image pixels in the image being greater thanthe plurality of sensor pixels.
 12. An imaging system, comprising: anarray of lenses; a plurality of sensor pixels for each lens; acorresponding plurality of filters, a filter provided for each sensorpixel having multiple sub-pixel resolution elements and providing asub-pixel shifted multiple image on each sensor pixel; and a processorreceiving outputs from each sensor pixel and reconstructing an image, anumber of image pixels in the image being greater than the plurality ofsensor pixels.
 13. The imaging system as recited in claim 12, furthercomprising a birefringent structure plurality of filters.
 14. Theimaging system as recited in claim 12, further comprising acorresponding plurality of focusing lenses, a focusing lens between thefilter and a corresponding sensor pixel.
 15. The imaging system asrecited in claim 12, wherein at least one sensor pixel receives lightfrom more than one lens of the array of lenses.
 16. The imaging systemas recited in claim 12, wherein the filter is an apertured mask.